Data mining technique with distributed novelty search

ABSTRACT

Roughly described, an evolutionary data mining system includes at least two processing units, each having a pool of candidate individuals in which each candidate individual has a fitness estimate and experience level. A first processing unit tests candidate individuals against training data, updates an individual&#39;s experience level, and assigns each candidate to one of multiple layers of the candidate pool based on the individual&#39;s experience level. Individuals within the same layer of the same pool compete with each other to remain candidates. The first processing unit selects a set of candidates to retain based on the relative novelty of their responses to the training data. The first processing unit reports successful individuals to the second processing unit, and receives individuals for further testing from the second processing unit. The second processing unit selects individuals to retain based on their fitness estimate.

CROSS-REFERENCE TO OTHER APPLICATION

U.S. Pat. No. 8,909,570, entitled “Data Mining Technique withExperience-layered Gene Pool,” issued on Dec. 9, 2014 is incorporated byreference herein.

BACKGROUND

The invention relates generally to improved genetic algorithms toextract useful rules or relationships from a data set for use incontrolling systems.

In many environments, a large amount of data can be or has beencollected which records experience over time within the environment. Forexample, a healthcare environment may record clinical data, diagnosesand treatment regimens for a large number of patients, as well asoutcomes. A business environment may record customer information such aswho they are and what they do, and their browsing and purchasinghistories. A computer security environment may record a large number ofsoftware code examples that have been found to be malicious. A financialasset trading environment may record historical price trends and relatedstatistics about numerous financial assets (e.g., securities, indices,currencies) over a long period of time. Despite the large quantities ofsuch data, or perhaps because of it, deriving useful knowledge from suchdata stores can be a daunting task.

The process of extracting patterns from such data sets is known as datamining. Many techniques have been applied to the problem, but thepresent discussion concerns a class of techniques known as geneticalgorithms. Genetic algorithms have been applied to all of theabove-mentioned environments. With respect to stock categorization, forexample, according to one theory, at any given time, 5% of stocks followa trend. Genetic algorithms are thus sometimes used, with some success,to categorize a stock as following or not following a trend.

Evolutionary algorithms, which are supersets of Genetic Algorithms, areclassifiers which are good at traversing chaotic search spaces.According to Koza, J. R., “Genetic Programming: On the Programming ofComputers by Means of Natural Selection”, MIT Press (1992), incorporatedby reference herein, an evolutionary algorithm can be used to evolvecomplete programs in declarative notation. The basic elements of anevolutionary algorithm are an environment, a model for a genotype(referred to herein as an “individual”), a fitness function, and aprocreation function. An environment may be a model of any problemstatement. An individual may be defined by a set of rules governing itsbehavior within the environment. A rule may be a list of conditionsfollowed by an action to be performed in the environment.

A fitness function may be defined by the degree to which an evolvingrule set is successfully negotiating the environment. A fitness functionis thus used for evaluating the fitness of each individual in theenvironment.

A procreation function generates new individuals by mixing rules of theparent individuals. In each generation, a new population of individualsis created.

At the start of the evolutionary process, individuals constituting theinitial population are created randomly, by putting together thebuilding blocks, or alphabets, that form an individual. In geneticprogramming, the alphabets are a set of conditions and actions making uprules governing the behavior of the individual within the environment.Once a population is established, a set of individuals within theenvironment are selected to create the next generation in a processcalled procreation. Through procreation, rules of parent individuals aremixed, and sometimes mutated (i.e., a random change is made in a rule)to create a new rule set. This new rule set is then assigned to a childindividual that will be a member of the new generation. In someincarnations, known as elitist methods, the fittest members of theprevious generation, called elitists, are also preserved into the nextgeneration.

A homogeneous population can cause a genetic algorithm to convergeprematurely to local optima. Once such convergence occurs, the searchtypically terminates without having found optima which may be moreglobal. Aspects of the present invention address this problem.

I. SUMMARY

U.S. patent application Ser. No. 14/011,062 (GNFN 3100-1), incorporatedby reference herein, describes client/server arrangements forimplementing an evolutionary data mining system. In some sucharrangements, the pool of candidate individuals is distributed over amultitude of clients for evaluation. Each client continues to evaluateits own client-centric candidate pool using portions of data from atraining database or data feed, which it may receive in bulk orrecurrently. Individuals that satisfy one or more predefined conditionson a client computer are transmitted to the server to form part of aserver candidate pool.

In the above-incorporated application, the functions of the server maybe federated. Roughly described, this is achieved by providing“mid-chain” evolutionary coordinators, and placing them between the mainserver (which in this arrangement can be called a “top-chain”evolutionary coordinator, or a “master” evolutionary coordinator) andthe clients (which in this arrangement can be called “evolutionaryengines”). Multiple levels of mid-chain evolutionary coordinators can beused in a hierarchy, and the various branches of the hierarchy need nothave equal length. Each evolutionary coordinator (other than thetop-chain evolutionary coordinator) appears to its up-chain neighbor asif it were an evolutionary engine, though it does not actually performany evolution itself. Similarly, each evolutionary coordinator(including the top-chain evolutionary coordinator) also appears to itsdown-chain neighbors as a top-chain evolutionary coordinator. Eachmid-chain evolutionary coordinator maintains its own local candidatepool, reducing the load on the top-chain evolutionary coordinator pool,as well as reducing bandwidth requirements. As used herein, the term“evolutionary unit” includes both “evolutionary coordinators” and“evolutionary engines”.

To achieve a diverse population, individuals may be selected forprocreation based on the diversity of their output with respect to acommon set of test data. A novelty function may evaluate the differencein behavior between a pair of individuals, allowing selection of a setof individuals that demonstrate the most diverse behavior for a commonset of inputs. For example, individuals producing outputs that are mostdifferent from each other are considered herein to be most novel withrespect to one another. On the other hand, individuals which produceoutputs that are most similar to one another are considered herein to beleast novel with respect to one another, even if their internal rulesare markedly different from each other.

Many application domains are best served by producing a diversity ofsolutions, covering the search space as broadly as possible. NoveltySearch, by its very nature, promotes phenotypical diversity. Inembodiments herein, Evolutionary Engines (EEs) can be configured to useNovelty Search for their parent selection, while the EvolutionaryCoordinators (ECs) make use of a fitness estimate to decide whichindividuals to allow into the server pool. Evolutionary Engines receiveindividuals from Evolutionary Coordinators for gaining additionalexperience, and these individuals are added to the elitist pool.

The above summary is provided in order to provide a basic understandingof some aspects of the invention. This summary is not intended toidentify key or critical elements of the invention or to delineate thescope of the invention. Its sole purpose is to present some concepts ofthe invention in a simplified form as a prelude to the more detaileddescription that is presented later. Particular aspects of the inventionare described in the claims, specification and drawings.

II. BRIEF DESCRIPTION OF THE DRAWINGS

The invention will be described with respect to specific embodimentsthereof, and reference will be made to the drawings, in which:

FIG. 1 is an overall diagram of an embodiment of a data mining systemincorporating features of the invention.

FIG. 2 is a symbolic drawing of the candidate pool in FIG. 1.

FIG. 3 is a symbolic drawing of an individual in either the candidatepool or the production population of individuals in FIG. 1.

FIG. 4 is a symbolic drawing indicating how training data is organizedin the training database in FIG. 1.

FIG. 5 illustrates various modules that can be used to implement thefunctionality of an evolutionary engine of FIG. 11.

FIG. 6 is a flow diagram showing the basic operation of eachevolutionary engine of the evolutionary mining system.

FIG. 7 is a flow diagram showing operations in a competition module ofan evolutionary engine, according to an embodiment of the invention.

FIG. 8 illustrates various modules that can be used to implement thefunctionality of an evolutionary coordinator in a client/serverconfiguration, according to an embodiment of the invention.

FIG. 9 is a flow diagram showing the basic operation of eachevolutionary coordinator of the evolutionary mining system, according toan embodiment of the invention.

FIG. 10 is a flow diagram illustrating a method of operation of thecompetition modules for an evolutionary coordinator, according to anembodiment of the invention.

FIG. 11 is a symbolic diagram of a federated training system of FIG. 1,according to an embodiment of the invention.

FIG. 12 illustrates various modules that can be used to implement thefunctionality of a mid-chain evolutionary coordinator of FIG. 11,according to an embodiment of the invention.

FIG. 13 illustrates various modules that can be used to implement thefunctionality of a top-chain evolutionary coordinator of FIG. 11,according to an embodiment of the invention.

FIG. 14 is a simplified block diagram of a computer system that can beused to implement any or all of the evolutionary units, the productionsystem, and the data feed server in FIGS. 1 and 11, according to anembodiment of the invention.

III. DETAILED DESCRIPTION

The following description is presented to enable any person skilled inthe art to make and use the invention, and is provided in the context ofa particular application and its requirements. Various modifications tothe disclosed embodiments will be readily apparent to those skilled inthe art, and the general principles defined herein may be applied toother embodiments and applications without departing from the spirit andscope of the present invention. Thus, the present invention is notintended to be limited to the embodiments shown, but is to be accordedthe widest scope consistent with the principles and features disclosedherein.

Data mining involves searching for patterns in a database. The fittestindividuals are considered to be those that identify patterns in thedatabase that optimize for some result. In embodiments herein, thedatabase is a training database, and the result is also represented insome way in the database. Once fit individuals have been identified,they can be used to identify patterns in production data which arelikely to produce the desired result. In a healthcare environment, theindividual can be used to point out patterns in diagnosis and treatmentdata which should be studied more closely as likely either improving ordegrading a patient's diagnosis. In a financial assets tradingenvironment, the individual can be used to detect patterns in real timedata and assert trading signals to a trading desk. The action signalsfrom an individual can be transmitted to the appropriate controlledsystem for execution.

One difference between the data mining environments of the embodimentsdescribed herein, and many other environments in which evolutionaryalgorithms can be applied, is that the fitness of a particularindividual in the data mining environment usually cannot be determinedby a single test of the individual on the data; rather, the fitnessestimation itself tends to vary as it is tested on more and more samplesin the training database. The fitness estimate can be inaccurate astesting begins, and confidence in its accuracy increases as testing onmore samples continues. This means that if an individual is “lucky”early on, in the sense that the first set of samples that it was givenfor testing happened to have been in some sense “easy”, then after onlythe first set of samples the individual will appear to be fitter than itactually is. If compared to other individuals that have much moreexperience, lucky individuals could displace individuals whose fitnessestimates are lower but more realistic. If care is not taken, therefore,the algorithm will optimize for individuals that are lucky early on,rather than their actual fitness.

A solution to the problem of optimizing for individuals that are luckyearly on, rather than their actual fitness, is to consider individualsfor the elitist pool only after they have completed testing on apredetermined number of samples, for example 1000 samples. Once anindividual has reached that minimum threshold experience level,comparisons with other individuals are considered valid and can competeon the basis of fitness for a place in the elitist pool. The sameproblem can occur to a lesser degree even to individuals within theelitist pool, and a similar solution can be applied there as well. Thusin general, in embodiments herein, the elitist pool contains T layersnumbered L₁-L_(T), with T>1. The overall pool of candidate individualsalso includes some that have not yet undergone sufficient numbers oftests to be considered for the elitist pool, and those individuals areconsidered herein to reside in a layer below the elitist pool, designedlayer 0 (L₀). Each i'th one of the layers in [L₀ . . . L_(T−1)] containsonly individuals with a respective range of testing experience[ExpMin(L_(i)) . . . ExpMax(L_(i))], each ExpMin(L_(1+i))>ExpMax(L_(i)).The minimum experience level of the bottom layer L₀ is 0, and the toplayer L_(T) has a minimum experience level ExpMin(L_(T)) but no maximumexperience level. Preferably, the experience ranges of contiguous layersare themselves contiguous, so that ExpMin(L_(i+1))=ExpMax(L_(i))+1, for0<=i<T. Note that testing experience level is a significantly differentbasis on which to stratify individuals in an elitist pool than age inthe sense of ALPS. ALPS means Age-Layered Population Structure, in whichan individual's age is used to restrict competition and breeding betweenindividuals in the population. In the parlance of ALPS, “age” is ameasure of the number of times that an individual's genetic material hassurvived a generation (i.e., the number of times it has been preserveddue to being selected into the elitist pool), rather than a measure ofthe number of training samples on which an individual has been tested.

In an embodiment, each layer i in the elitist pool (i.e. in layers [L₁ .. . L_(T)]) is permitted to hold a respective maximum number ofindividuals, Quota(L_(i)). The quota is chosen to be small enough toensure competition among the individuals within the corresponding rangeof experience levels, but large enough to ensure sufficient diversityamong the fit individuals that graduate to the next higher layer.Preferably the quota of each such layer is fixed, but in anotherembodiment it could vary. The quota of layer L₀ is not chosen based onthese criteria, since the individuals in that layer do not yet compete.Preferably the number of layers T in the elitist pool is also fixed, butin another embodiment it can vary.

As each individual gains more experience, assuming it is not displacedwithin its current experience layer, it will eventually graduate to thenext higher experience layer. If the next higher experience layer is notyet full, then the individual may be added to that higher layer. If thehigher layer is full, then the individual has to compete for its placein that layer. If the graduating individual does not successfullycompete in the new layer, then the individual is discarded, and theindividuals in the next higher layer will be retained. Otherwise, thegraduating individual is accepted into the higher layer and a displacedindividual is discarded. The algorithm for performing competition amongindividuals in the elitist pool may be different in an evolutionaryengine than in an evolutionary coordinator. In an embodiment,individuals that have reached the top layer of the EC candidate pool donot undergo further testing.

In the client/server arrangement, each client evaluates its ownclient-centric candidate pool using portions of data from a trainingdatabase or data feed, and individuals that satisfy one or morepredefined conditions on a client computer are transmitted to the serverto form part of a server candidate pool. The operation of anevolutionary engine (client) and evolutionary coordinator (server) havemany similarities. Each type of evolutionary unit receives individuals,assigns each incoming individual to a layer in its local candidate pool,discards individuals in layers that exceed a quota for the number ofindividuals for that layer, and identifies individuals to pass outsideof the unit including: the client harvesting the best individuals tosend to the server, the server sending individuals back to a client forfurther testing, or the server passing individuals into production.

FIG. 1 is an overall diagram of an embodiment of a data mining systemincorporating features of the invention. The system is divided intothree portions, a training system 110, a production system 112, and acontrolled system 128. The training system 110 interacts with a database114 containing training data, as well as with another database 116containing the candidate pool. As used herein, the term “database” doesnot necessarily imply any unity of structure. For example, two or moreseparate databases, when considered together, still constitute a“database” as that term is used herein. In particular, though candidatepool 116 may appear in FIG. 1 as a unitary structure, whereas in thefederated embodiments described herein it is actually spread overmultiple storage engines. The candidate pool database 116 includes aportion 118 containing the elitist pool. The training system 110operates according to a fitness function 120, which indicates to thetraining system 110 how to measure the fitness of an individual. Thetraining system 110 optimizes for individuals that have the greatestfitness, however fitness is defined by the fitness function 120. Thefitness function is specific to the environment and goals of theparticular application. For example, the fitness function may be afunction of the predictive value of the individual as assessed againstthe training data—the more often the individual correctly predicts theresult represented in the training data, the more fit the individual isconsidered. In a financial asset trading environment, an individualmight provide trading signals (e.g. buy, sell, hold current position,exit current position), and fitness may be measured by the individual'sability to make a profit, or the ability to do so while maintainingstability, or some other desired property. In the healthcare domain, anindividual might propose a diagnosis based on patient prior treatmentand current vital signs, and fitness may be measured by the accuracy ofthat diagnosis as represented in the training data.

The production system 112 operates according to a production populationof individuals in another database 122. The production system 112applies these individuals to production data 124, and produces outputs126, which may be action signals or recommendations. In the financialasset trading environment, for example, the production data 124 may be astream of real time stock prices and the outputs 126 of the productionsystem 112 may be the trading signals or instructions that one or moreof the individuals in production population 122 outputs in response tothe production data 124. In the healthcare domain, the production data124 may be current patient data, and the outputs 126 of the productionsystem 112 may be a suggested diagnosis or treatment regimen that one ormore of the individuals in production population 122 outputs in responseto the production data 124. The production population 122 is harvestedfrom the training system 110 once or at intervals, depending on theembodiment. Preferably, only individuals from elitist pool 118 arepermitted to be harvested. In some embodiments, only individuals in thetop layer of the elitist pool are harvested. In an embodiment, furtherselection criteria is applied in the harvesting process.

The controlled system 128 is a system that is controlled automaticallyby the signals 126 from the production system. In the financial assettrading environment, for example, the controlled system may be a fullyautomated brokerage system which receives the trading signals via acomputer network (not shown in FIG. 1) and takes the indicated action.Depending on the application environment, the controlled system 128 mayalso include mechanical systems such as engines, air-conditioners,refrigerators, electric motors, robots, milling equipment, constructionequipment, or a manufacturing plant.

FIG. 2 is a symbolic drawing of the candidate pool 116 in FIG. 1. As canbe seen, the individuals in the pool are stratified into T+1 “experiencelayers”, labeled L₀ through L_(T). The individuals in L₀ are veryinexperienced (have been tested on only a relatively small number ofsamples in training data 114, if any), whereas the higher layers containindividuals in successively greater experience ranges. The layers L₁through L_(T) constitute the elitist pool 118 (FIG. 1). Each layer i inthe elitist pool 118 has associated therewith three “layer parameters”:a quota Quota(L_(i)) for the layer, a range of experience levels[ExpMin(L_(i)) . . . ExpMax(L_(i))] for the layer, and in someembodiments the minimum fitness FitMin(L_(i)) for the layer. Forexample, an embodiment in the financial asset trading environment mayhave on the order of 40 or 50 layers in the elitist pool, eachcontaining individuals with experience levels within a range on theorder of 4000-5000 trials. The minimum experience level ExpMin(L₁) maybe on the order of 8000-10,000 trials, and each layer may have a quotaon the order of 100 individuals.

In the embodiment of FIG. 2, the quotas for all the layers in theelitist pool 118 are equal and fixed. In another embodiment, the quotasare not required to be fixed, nor are the quotas required to be the sameacross layers of the candidate pool. In addition, ExpMin(L₀)=0 in thisembodiment. Also, as the experience ranges of the layers are contiguous,ExpMin of each layer can be inferred as one higher than ExpMax of thenext lower layer, or ExpMax of each layer can be inferred as one lowerthan ExpMin of the next higher layer. Thus only the minimum experiencelevel or the maximum experience level need be specified for each layer.In the embodiment, only the minimum experience levels are specified, andthey are specified for layers L₁-L_(T); in another embodiment only themaximum experience levels are specified, and they are specified forlayers L₀-L_(T−1). In yet another embodiment, the size of the range ofexperience layers assigned to all the layers is constant, and only oneminimum or maximum experience level is specified in the layerparameters; the remainder are calculated algorithmically as needed.Other variations will be apparent.

For an embodiment in which each layer is assigned a minimum fitnessvalue, the FitMin( ) values associated with layers of the elitist poolare not specified a priori. Rather, they are filled by copying from thefitness estimate associated with the least fit individual in each layer.Whenever the fitness estimate of the least fit individual is updated,and whenever the least fit individual itself is replaced, the FitMin( )value associated with the layer is updated correspondingly. The FitMin() values are needed for comparing to the fitness estimation ofindividuals coming up from the next lower layer, and having themassociated directly with each layer can simplify this comparison. Inanother embodiment, each layer can instead contain a pointer to theleast fit individual in the layer, and the comparison method can obtainthe layer minimum fitness from that individual itself. In general, eachlayer has associated with it an “indication” of the minimum fitness inthe layer. As used herein, an “indication” of an item of informationdoes not necessarily require the direct specification of that item ofinformation. Information can be “indicated” in a field by simplyreferring to the actual information through one or more layers ofindirection, or by identifying one or more items of differentinformation which are together sufficient to determine the actual itemof information. In addition, the term “identification” and its variantsare used herein to mean the same as “indication”.

In one embodiment, the experience layers in candidate pool 116 defineseparate regions of memory, and the individuals having experience levelswithin the range of each particular layer are stored physically withinthat layer. Preferably, however, the experience layers are only impliedby the layer parameters and the individuals can actually be locatedanywhere in memory. In one embodiment, the individuals in candidate pool116 are stored and managed by conventional database management systems(DBMS), and are accessed using SQL statements. Thus a conventional SQLquery can be used to obtain, for example, the fitness estimate of theleast fit individual in the highest layer. New individuals can beinserted into the candidate pool 116 using the SQL “insert” statement,and individuals being discarded can be deleted using the SQL “delete”statement. In another embodiment, the individuals in candidate pool 116are stored in a linked list. In such an embodiment insertion of a newindividual can be accomplished by writing its contents into an elementin a free list, and then linking the element into the main linked list.Discarding of individuals involves unlinking them from the main linkedlist and re-linking them into the free list. Discarding causes anindividual to be removed from competition, but in some embodiments,information about the individual may be recorded or logged for otherpurposes.

FIG. 3 is a symbolic drawing of an individual 310 in either thecandidate pool 116 or the production population 122 of individuals. Asused herein, an “individual” is defined by its contents. An individualcreated by procreation is considered herein to constitute a differentindividual than its parents, even though it retains some if its parents'genetic material. In this embodiment, the individual identifies an ID312, its experience level 314, and its current fitness estimate 316. Anindividual represents a full solution-space in that it contains the“classification rules” 318 needed to classify an item of test data. Inthis embodiment, each row of the table represents a classification rule.Each rule contains one or more conditions 320 and an output 322 to beasserted if all the conditions in a given sample are true. Duringprocreation, any of the conditions or any of the outputs may be altered,or even entire rules may be replaced. The individual's experience level314 increments by one for each sample of the training data 114 on whichit is tested, and its fitness estimate 316 is determined by fitnessfunction 120, averaged (or otherwise combined) over the all the trials.

A rule is a conjunctive list of indicator-based conditions inassociation with an output. Indicators are the system inputs that can befed to a condition. These indicators are represented in the trainingdatabase 114, as well as in the production data 124. Indicators can alsobe introspective, for example by indicating the fitness estimate of theindividual at any given moment. In the embodiment of FIG. 3, theindividual's conditions are all specified as parameter/value (“P/V”)pairs, where each P is a parameter, and each V is a value for theparameter. For example, a “parameter” in one of the conditions might be“minimum time of day”, and the “value” of the condition might be “3:00pm”. In this case the condition is satisfied for a given sample of dataonly if the time-of-day stamp on the sample (the indicator) has a valueof 3:00 pm or later. Another embodiment can also include conditionswhich are themselves conditioned on other items (such as otherconditions in the rule or in a different rule or the result of anotherentire one of the rules). Yet another embodiment can also includeconditions or rules which are specified procedurally rather than as P/Vpairs. Many other variations will be apparent. In the drawing of FIG. 3,an individual is illustrated as having a rectangular table of rules, arule being illustrated as a horizontal row of the table. Each conditionof the rule is illustrated as a cell in the row. While the drawing showseach of the rules as having the same number M of conditions, this isonly a simplification for convenience of description, because theyenable the drawing to identify individual conditions in the form “P/Vx,y”, where x is the rule (row) number and y is the condition (column)number within rule x. In general different rules in an individual canhave different numbers of conditions.

In a financial asset trading embodiment, during training, an individualcan be thought of as a virtual trader that is given a hypothetical sumof money to trade using historical data. Such trades are performed inaccordance with a set of rules that define the individual therebyprompting it to buy, sell, hold its position, or exit its position. Theoutputs of the rules are trading action signals or instructions, such asbuy, sell, exit or hold. Rules may also be designed to contain gain-goaland stop-loss targets, thus rendering the exit action redundant. A holdoccurs when no rule in the individual is triggered, therefore, theindividual effectively holds its current position. The indicators onwhich the rules are based can be, for example, a time increment(“tick”), or the closing price for a stock day.

The following code defines an example rule in terms of conditions andindicators, as well as the action asserted by the rule, in accordancewith one embodiment of the present invention:

if (PositionProfit>=2% and !(tick=(−54/10000)% prev tick and MACD isnegative)

and !(tick=(−119/10000)% prev tick and Position is long))

and !(ADX×100<=5052))

then SELL

where “and” represents logical “AND” operation, “!” represents logical“NOT” operation, “tick”, “MACD” and “ADX” are stock indicators, “SELL”represents action to sell, and “PositionProfit” represents the profitposition of the individual.

In a healthcare embodiment, an individual can be thought of as a set ofrules predicting a patient's future state, given the patient's currentand past state. The outputs of the rules can be proposed diagnoses orproposed treatment regimens that the individual asserts are appropriategiven the conditions of the individual's rules. The indicators on whichthe rules are based can be a patient's vital signs, and past treatmentand medication history, for example. An example rule is as follows:

-   -   if pulse>=120 and 18<=blood pressure[6]<20 and temp>=104 and        surgery duration<22 and clamp on artery and medication=EB45 and        last medication>=60 and !white blood cell count [3]<−2.3 and        !oxygen level [1]<−1.1-->>>    -   then thromboembolism @prob<=0.65

The training data is arranged in the database 114 as a set of samples,each with parameters and their values, as well as sufficient informationto determine a result that can be compared with an assertion made by anindividual on the values in the sample. In one embodiment, the result isexplicit, for example a number set out explicitly in association withthe sample. In such an embodiment, the fitness function can be dependentupon the number of samples for which the individual's output matches theresult of the sample. In another embodiment, such as in the financialasset trading embodiment, the result may be only implicit. For example,the sample may include the price of an asset at each tick throughout atrading day, and the training system 110 must hypothetically perform allthe trading recommendations made by the individual throughout thetrading day in order to determine whether and to what extent theindividual made a profit or loss. The fitness function can be dependentupon the profit or loss that the individual, as a hypothetical trader,would have made using the tick data for the sample.

FIG. 4 is a symbolic drawing indicating how the training data isorganized in the database 114. The illustration in FIG. 4 is for thefinancial asset trading embodiment, and it will be understood how it canbe modified for use in other environments. Referring to FIG. 4, threesamples 410 are shown. Each sample includes a historical date, anidentification of a particular security or other financial asset (suchas a particular stock symbol), and raw historical market data for thatfinancial asset on that entire trading day, e.g. tick data, tradingvolume data, price, etc.; and all other data needed to test performanceof the individual's trading recommendations on this asset on thishistorical trading day. In another embodiment, a sample can contain tickdata for a different time interval, which may be shorter or longer thanone trading day.

Federated Client/Server Arrangement

An embodiment of the invention will be described in the context of afederated arrangement.

In some environments, the training data used to evaluate an individual'sfitness can be voluminous. Therefore, even with modern high processingpower and large memory capacity computers, achieving quality resultswithin a reasonable time is often not feasible on a single machine. Alarge candidate pool also requires a large memory and high processingpower. In one embodiment, therefore, a federated client/server model isused to provide scaling in order to achieve high quality evaluationresults within a reasonable time period.

Roughly described, the federated arrangement is achieved by providing“mid-chain” evolutionary coordinators, and placing them between the mainserver (which in this arrangement can be called a “top-chain”evolutionary coordinator, or a “master” evolutionary coordinator) andthe clients (which in this arrangement can be called “evolutionaryengines”). Multiple levels of mid-chain evolutionary coordinators can beused in a hierarchy, and the various branches of the hierarchy need nothave equal length. Each evolutionary coordinator (other than thetop-chain evolutionary coordinator) appears to its up-chain neighbor asif it were an evolutionary engine, though it does not actually performany evolution itself. Similarly, each evolutionary coordinator(including the top-chain evolutionary coordinator) also appears to itsdown-chain neighbors as a top-chain evolutionary coordinator. Eachmid-chain evolutionary coordinator maintains its own local candidatepool, reducing the load on the top-chain evolutionary coordinator pool,as well as reducing bandwidth requirements.

In broad overview, all the work in testing of candidate individuals ontraining data is performed by the evolutionary engines. The EE's alsogenerate their own initial sets of individuals, enforce competitionamong the individuals in their own respective candidate pools, andevolve their best performing candidates by procreation. The EC's, on theother hand, perform no testing. Instead they merely coordinate theactivities of their respective down-chain engines. Each evolutionaryengine that has an up-chain neighbor ends its best performing candidatesto its up-chain EC, and also receives additional candidates from itsup-chain EC for further testing. Each evolutionary coordinator that hasa down-chain neighbor (i.e. each EC in FIG. 11) receives individualsfrom its respective down-chain engines which the down-chain engines hadconsidered top performers, and requires the received individuals tocompete for entry into the EC's own local candidate pool 1122. If areceived candidate is one which the EC had previously sent down to thedown-chain engine for further testing, then the EC first updates itslocal understanding of the fitness of the individual prior to thecompetition. Each EC also sends down candidates from its own local poolfor further testing as required. At various times, like the EE's, eachEC harvests individuals from its own local pool which the EC considersto be its top performers. If the EC is a mid-chain EC 1120, then itsends its harvested individuals to its up-chain EC, which may be thetop-chain EC 1110 or another mid-chain EC 1120. If the EC is thetop-chain EC, then it sends its harvested individuals for deployment.

In a federated environment, an evolutionary engine need not be aware ofwhether its immediate coordinator is a top-level coordinator, as in theclient/server case described earlier, or whether its immediatecoordinator is a mid-chain coordinator. Similarly, the top-levelcoordinator need not be aware that it directly coordinates mid-chaincoordinators rather than coordinating evolutionary engines directly.Thus, a mid-chain coordinator operates as a coordinator as describedabove with the following exceptions. The individuals harvested from amid-chain coordinator are not placed directly into production, butrather, are forwarded to a higher level coordinator, such as thetop-level coordinator. Also, a mid-chain coordinator may receiveindividuals from its up-chain coordinator, the individuals requiringfurther testing. Except for these two exceptions, a mid-chaincoordinator behaves as an evolutionary engine.

FIG. 11 is a symbolic diagram of training system 110. It comprises atop-chain evolutionary coordinator (EC) 1110, which is also sometimesreferred to herein as the master EC. Top-chain EC 1110 maintains themaster candidate pool 1112.

Down-chain from the top-chain EC 1110 is a set of mid-chain EC's 1120-1through 1120-6 (collectively 1120). Specifically, mid-chain EC's 1120-1through 1120-3 are immediately down-chain from top-chain EC 1110.Mid-chain EC 1120-4 is immediately down-chain from mid-chain EC 1120-2,and mid-chain EC's 1120-5 and 1120-6 are each immediately down-chainfrom mid-chain EC 1120-3. Each of the mid-chain EC's 1120 maintains itsown local candidate pool 1122-1 through 1122-6, respectively(collectively 1122).

Down-chain from the mid-chain EC's 1120 are a plurality of evolutionaryengines (EE's) 1130-1 through 1130-9 (collectively 1130). Specifically,EE 1130-1 is immediately down-chain from top-chain EC 1110, and EE's1130-2 and 1130-3 are each immediately down-chain from mid-chain EC1120-1. EE 1130-4 is immediately down-chain from mid-chain EC 1120-2,and EE's 1130-5 and 1130-6 are each immediately down-chain frommid-chain EC 1120-4. EE's 1130-7 and 1130-8 are each immediatelydown-chain from mid-chain EEC 1120-5, and EE 1130-9 is immediatelydown-chain from mid-chain EEC 1120-6. Like the EC's 1120, each of theEE's 1130 maintains its own local candidate pool 1132-1 through 1132-6,respectively (collectively 1132).

Each EE 1130 further has a communication port through which it canaccess one or more data feed servers 1140, which retrieve and forwardtraining samples from the training database 114. Alternatively, althoughnot shown, the training samples may be supplied from data feed server1140 to the EE's 1130 via one or more of the EC's 1120. The data feedserver 1140 can also be thought of as simply a port through which thedata arrives or is retrieved. Each of the EC's 1110 and 1120 maintains alocal record of the IP address and port number at which each of itsimmediate down-chain engines receives individuals delegated forevaluation, and delegating an individual to a particular one of thedown-chain engines for evaluation involves transmitting the individual(or an identification of the individual) toward the IP address and portnumber of the particular engine.

The EE's 1130, and in some embodiments one or more of the EC's 1120 aswell, are volunteers in the sense that they can come and go withoutinstruction from the up-chain neighboring engines. When an EC 1120 joinsthe arrangement, it receives the IP address and port number of itsimmediately up-chain neighbor and the minimum experience levelacceptable to the up-chain neighbor for candidates being sent up fromthe new EC 1120. EE's 1130 joining the arrangement receive thatinformation plus the IP address and port number of data feed server1140. This information can be sent by any server that manages thehierarchy of evolutionary engines in the system. In one embodiment thatcan be the top-chain evolutionary coordinator 1110, whereas in anotherembodiment it can be a separate dedicated management server (not shown).

As used herein, the terms down-chain and up-chain are complimentary: ifa second engine is down-chain from a first engine, then the first engineis up-chain from the second engine, and vice-versa. In addition, theterms “immediately” up-chain and “immediately” down-chain preclude anintervening evolutionary engine, whereas the terms up-chain anddown-chain themselves do not. Even “immediately”, however, does notpreclude intervening components that are not evolutionary engines. Alsoas used herein, the term “evolutionary engine” includes bothevolutionary coordinators and evolutionary engines, and the term“evolutionary coordinator” includes both mid-chain evolutionarycoordinators and the top-chain evolutionary coordinator.

It can be seen from FIG. 11 that the branches of the hierarchy of EC'scan be very non-uniform in length and spread. A given branch can containas few as zero mid-chain EC's 1120, or as many as ten or more in a givenembodiment. A given EC also can support as few as one down-chain engineor as many as ten or more, and some can be EE's 1130 while others areother mid-chain EC's 1120. This flexibility is facilitated by the rulethat each evolutionary engine appears to its immediately up-chainneighboring engine, if it has one, as if it were an evolutionary engine;and appears to its immediately down-chain neighboring engines, if it hasany, as if it were a top-chain evolutionary coordinator.

Moreover, each of the evolutionary engines in FIG. 11 can itself be acluster of machines rather than just one. It can also be physical orvirtual or, in the case of a cluster, partially physical and partiallyvirtual. As a cluster, an evolutionary engine still appears to itsup-chain and down-chain engines as a single evolutionary engine orevolutionary coordinator as desired, so that the up-chain and down-chainengines do not need to know that it is not a simple computer system. Forexample, one of the mid-chain coordinators can itself be made up of itsown internal hierarchy of a top-chain coordinator and one or moremid-chain coordinators, thereby forming a nested arrangement. Similarly,an evolutionary engine can be made up of its own internal hierarchy ofengines, such as an internal top-chain coordinator up-chain of one ormore internal evolutionary engines, with or without a level of internalmid-chain coordinators.

Still further, in the embodiment of FIG. 11, each evolutionary enginehas only one immediately up-chain engine. This is so that when an engineharvests an individual and forwards it up-chain, it will not improperlyreturn the individual to an up-chain engine different from the one thatdelegated it. Another embodiment may have no such restriction, allowinga given evolutionary engine to have more than one immediately up-chainengine. For example, this might be accomplished by associating, witheach individual delegated to another engine in the hierarchy, anindication of the engine to which it should be returned after testing.For new individuals created by an engine having more than oneimmediately up-chain engine (or created by an engine down-chain from anengine having more than one immediately up-chain engine), thearrangement can implement some predetermined algorithm (e.g. a singledefault, round robin, or random) for determining to which up-chainengine the individual should be sent after testing and harvesting.Numerous additional variations will be apparent to the reader.

In the arrangement of FIG. 11, scaling is carried out in two dimensions,namely in pool size as well as in evaluation of the same individual togenerate a more diverse candidate pool so as to increase the probabilityof finding fitter individuals. The candidate pool is distributed over amultitude of EE's 1130 for evaluation. Each EE evaluates its own localcandidate pool using data from training database 114, and individualsthat satisfy one or more predefined conditions on an EE 1130 aretransmitted up-chain to form part of the candidate pool in its up-chainEC.

Distributed processing of individuals also may be used to increase thespeed of evaluation of a given individual. To achieve this, individualsthat are returned to an EC after some testing, but additional testing isdesired, may be sent back (delegated) from the EC to a multitude ofdown-chain engines for further evaluation. The evaluation resultachieved by the down-chain engines (sometimes referred to herein aspartial evaluation) for an individual is transferred back to thedelegating EC. The EC merges the partial evaluation results of anindividual with that individual's fitness estimate at the time it wasdelegated to arrive at an updated fitness estimate for that individualas regards the EC's local candidate pool. For example, assume that anindividual has been tested on 1100 samples and is sent from a particularEC to, for example, two down-chain engines (which may be an EE 1130 oranother mid-chain EC 1122, or one of each), each instructed to test theindividual on 100 additional samples. Each of the down-chain enginesfurther tests the individual on the additional 100 samples (themid-chain EC 1120 further delegating that task to its further down-chainengines), and reports its own view of the fitness estimate to therequesting up-chain particular EC. The particular EC, having receivedback the individual with the requested additional testing experience,combines these two estimates with the individual's fitness estimate atthe time it was sent to the two down-chain engines, to calculate anupdated fitness estimate for the individual as viewed by the particularEC. The combined results represent the individual's fitness evaluatedover 700 days. In other words, the distributed system, in accordancewith this example, increases the experience level of an individual from1100 samples to 700 samples using only 100 different training samples ateach evolutionary engine. A distributed system, in accordance with thepresent invention, is thus highly scalable in evaluating itsindividuals.

In an embodiment, the top-chain EC 1110 maintains locally the mastercandidate pool. It is experience layered as in FIG. 2, but it does notmaintain any candidate individuals below its layer L₁. New individualsare created by evolutionary engines 1130, and they are not reported tothe top-chain EC 1110 until they have been tested on sufficient numbersof samples to qualify for the elitist pool 118 of the top-chain EC 1110.

Advantageously, EE's 1130 are enabled to perform individual procreationlocally, thereby improving the quality of their individuals. Each EE1130 is a self-contained evolution device, not only evaluating theindividuals in its own pool, but also creating new generations ofindividuals and moving the evolutionary process forward locally. ThusEE's 1130 maintain their own local candidate pool which need not matcheach other's or that of any of the ECs. Since the EE's 1130 continue toadvance with their own local evolutionary process, their processingpower is not wasted even if they are not in constant communication withtheir up-chain neighbors. Once communication is reestablished with theup-chain neighbors, EE's 1130 can send in their fittest individualsup-chain and receive additional individuals from their up-chainneighbors for further testing.

Evolutionary Engine (EE)

FIG. 5 illustrates various modules that can be used to implement thefunctionality of an evolutionary engine 1130. The EE's local candidatepool 1132 is also shown in the drawing. Generally, solid lines indicateprocess flow, and broken lines indicate data flow. The modules can beimplemented in hardware or software, and need not be divided up inprecisely the same blocks as shown in FIG. 5. Some can also beimplemented on different processor cores or computers, or spread among anumber of different processors or computers. In addition, it will beappreciated that some of the modules can be combined, operated inparallel or in a different sequence than that shown in FIG. 5 withoutaffecting the functions achieved. Also as used herein, the term “module”can include “sub-modules”, which themselves can be considered herein toconstitute modules. In particular, the candidate testing module 512,competition module 514, and procreation module 516 are also consideredherein to be sub-modules of a candidate pool processing module 520. Theblocks in FIG. 5 designated as modules can also be thought of asflowchart steps in a method. These comments also apply to FIGS. 12 and13.

A candidate pool 1132 is local to one evolutionary engine and not sharedwith other evolutionary units. Thus, the same individual may reside inmultiple local candidate pools having different state (i.e. the view ofan individual may be different across engines).

EE—Receiving Individuals

FIG. 6 is a flow diagram showing the basic operation of an evolutionaryengine. The basic flow includes in Step 610 receiving new individualsinto the engine's candidate pool. Candidates are received into the localcandidate pool both by creation locally, and by insertion from anup-chain evolutionary coordinator. Referring to FIG. 5, the localcandidate pool 1132 is initialized by pool initialization module 510,which creates an initial set of candidate individuals in L₀ of thecandidate pool 1132. These individuals can be created randomly, or bysome other algorithm, or in some embodiments a priori knowledge is usedto seed the first generation. In another embodiment, individuals fromprior runs can be borrowed to seed a new run. At the start, allindividuals are initialized with an experience level of zero and afitness estimate that is undefined (not yet estimated).

The number of individuals created by the EE's 1130 may vary depending onthe memory size and the CPU processing power of the EE's. An EE 1130 maybe implemented on, in addition to the implementation options mentionedabove, a laptop computer, a desktop computer, a cellular/VoIP handheldcomputer or smart phone, a tablet computer, distributed computer, or thelike. An example system may have hundreds of thousands of EE's 1130, andan EE 1130 may have on the order of 1000 individuals for evaluation.Each of the evolutionary engines also includes a Candidate InsertionModule 522 by which Individuals may also be received for further testingfrom an evolutionary coordinator (“coordinator” or “EC”).

A newly received incoming individual includes at least an experiencelevel and a fitness estimate, though the fitness estimate may beundefined for newly created individuals. The individuals received from acoordinator for further testing retain their experience level andfitness estimates as received from the coordinator. New individuals donot compete with the other individuals in the local candidate pool 1132until after a battery of trials (which further refines their fitnessestimates and increases their experience levels prior to thecompetition). New individuals created by procreation within theprocreation module 516 may be directly inserted into the local candidatepool 1132 and/or provided to candidate testing module 512.

EE—Testing

In Step 620, individuals in the candidate pool are tested against sampledata. Candidate testing module 512 tests all individuals in the localcandidate pool. Each individual undergoes a battery of tests or trialson the training data 114, each trial testing the individual on onesample 410 that may represent a plurality of data points in an orderedcollection. For example they may represent data collected at multipletime points over the span of some time period such as a single day. Theindividual's experience level is then increased by the number of datasamples in the battery. The final result from testing against a sample(i.e. all data points in the sample) is an indication of the fitness ofthe individual with respect to that sample.

In an embodiment, the fitness estimate may be an average of the fitnessscore assigned to each of the results of all trials of the individual.In this case the fitness estimate can conveniently be indicated by twonumbers: the sum of the results of all trials of the individual, and thetotal number of trials that the individual has experienced. The latternumber may already be maintained as the experience level of theindividual. The fitness estimate at any particular time can then becalculated by dividing the sum of the results by the experience level ofthe individual. In an embodiment such as this, “updating” of the fitnessestimate can involve merely adding the results of the most recent trialsto the prior sum. It will be appreciated that the fitness estimatemaintained in the local candidate pool represents the individual'sfitness as viewed by the current evolutionary engine. If the individualhad been received from a coordinator for further testing, then thecoordinator's view of the individual's fitness may be different from theview of the evolutionary engine. It is for this reason that fitness issometimes referred to herein as being a fitness version that is“centric” to one evolutionary unit or another (i.e. as viewed by thatevolutionary unit).

The final result from testing against a sample is an indication of thefitness of the individuals with respect to that sample, but ‘how itreached that output’ is the behavior of the individual with respect tothat sample. For example, if the data points collected in a samplerepresent the share price of a particular stock, the individual'sbehavior in response to each data point of the sample may be to buy,sell, or hold. During this testing phase, each individual may be testedagainst multiple samples, and the test results of the individual foreach sample are recorded. The “behavior” of an individual relative toone sample then is given by its complete history of outputs (buy, sellor hold) for all data points in the sample. And the “behavior” of theindividual in a battery of tests is given by the individual's completehistory of outputs for all data points in all the samples in thebattery.

Because recording the complete behavior of an individual can require alot of memory, and can involve extensive computations in the competitionphase (described elsewhere herein), in an embodiment only a snapshot ofthe behavior of each individual is recorded. In one embodiment thesnapshot aggregates the behavior of the individual over all data pointsof a sample, so that only one behavioral value is recorded for each fullsample on which the individual is tested. This still produces onebehavioral value for the individual per sample in the battery of trials.In another embodiment the snapshot also aggregates the behavior of theindividual over all samples of a battery of trials, so that only onebehavioral value is recorded for the individual for the entire batteryof trials. Other kinds of snapshots are also possible in differentembodiments, such as aggregation of behavior across some but not alldata points in each sample. In an embodiment the behavior (or behavioralsnapshot) of an individual for a battery of tests is stored in a vector,with each vector entry storing a snapshot of the behavior of theindividual for one particular sample.

Other ways of representing individuals' behavior may be used if there isan associated difference function that takes the representation ofbehavior for two individuals and returns a distance indicator that canbe ranked against other such distance indicators in order to provide ameasure of the similarity of behavior of various pairs of individuals.

In some embodiments it may not be revealing to use the state of theindividual only at the end of an entire sample, as the aggregatedbehavioral snapshot for the individual over the entire sample. In suchembodiments it may be more revealing to define the aggregated behavioralsnapshot for an individual over an entire sample as merely the snapshotat one randomly chosen point within the ordered collection of datapoints of the sample. In this embodiment the point chosen to representthe behavior of an individual over a particular data sample is the samepoint for all individuals. In another embodiment, the snapshot may betaken at two points in the ordered collection, or some number k ofpoints less than the total number of points.

EE—Assign Individuals to Layer in the Candidate Pool

Competition module 514 assigns individuals to a layer in the localcandidate pool and determines which individuals to discard. In Step 630,individuals are assigned to a layer within the evolutionary engine'scandidate pool based on the value of their experience indicator. Higherlayers of the candidate pool are reserved for individuals that have beentested the most and survived elimination for each round of testing thusfar. Different embodiments use slightly different mechanisms forassigning individuals to layers of the candidate pool for anevolutionary engine. In one embodiment, the process of assigningindividuals to candidate pool layers can be thought of as occurring oneindividual at a time, as follows. First, a loop is begun through allindividuals in the local candidate pool whose experience level haschanged since the last time the competition module was executed. If thecurrent individual's experience level has not increased sufficiently toqualify it for the next experience layer in the candidate pool, then theindividual is ignored and the next one is considered. If the currentindividual's experience level has increased sufficiently to qualify itfor a new experience layer, then the competition module determineswhether the target experience layer is already at quota. If not, thenthe individual is simply moved into that experience level. If the targetlayer is full, then the competition module determines which individualswithin the layer to discard.

The above routine processes individuals sequentially, and differentembodiments can implement different sequences for processing theindividuals. Note that the processing sequence can affect the resultsif, for example, an individual in layer L_(i) is being considered forlayer L_(i+1) at the same time that an individual in layer L_(i−1) isbeing considered for layer L_(i). If the former test occurs first, thena hole will be opened in layer L_(i) and the individual graduating fromlayer L_(i−1) will be promoted into layer L_(i) automatically. If thelatter test occurs first, then the individual graduating from layerL_(i−1) will have to compete for its place in layer L_(i) (assuminglayer L_(i) is at quota). In another embodiment, individuals areconsidered layer by layer either according to their target layer afterpromotion, or according to their current layer prior to promotion.Again, the sequence of individuals to consider within each layer willdepend on the embodiment, as will the sequence in which the layersthemselves are considered.

In another embodiment, all individuals may be grouped by experiencelevel, then in a separate step all layers that are oversubscribed selectindividuals for discarding so that each layer retains no more than itsquota. In this embodiment, the order of assigning individuals to layersdoes not matter.

EE—Discarding Individuals

When the number of individuals assigned to a layer exceeds the quota forthat layer, then individuals in the layer need to compete to stay in thecandidate pool and avoid being discarded. In Step 640, individuals ineach layer of the candidate pool are selected for discarding. Multipleembodiments exist for selecting individuals for discarding, anddifferent evolutionary engines can implement different competitionalgorithms. FIG. 7 is a flow diagram illustrating a way for selectingindividuals to be discarded, according to an embodiment of theinvention. In an experience-layered embodiment, the discarding processis performed on each layer of the candidate pool independently of theother layers. Rather than select the fittest individuals, individualsare selected for discarding whose behavior is most similar to anotherindividual in the layer. In other words, a novelty search process isused to retain the individuals having the most diverse behavior, andthose individuals with redundant behavior are discarded.

Discarding causes an individual to be removed from competition, butinformation about the individual may be recorded or logged for otherpurposes.

EE—Selecting for Novelty

In the embodiment of FIG. 7, attempting to maximize the diversity ofindividuals within a layer of the candidate pool involves retaining asubset of the layer population, the subset having a number ofindividuals equal to the quota for the layer, and the individuals havingbehavior that is most dis-similar from other individuals in the subset.To achieve that, individuals having the most similar behavior areremoved from the layer until only the number of individuals equal to thequota are left in the layer. Within a single layer of the candidatepool, in step 720, a list of all pairs of individuals is generated. Foreach pair of individuals, a distance measure between them is determined.Determining a distance between individuals in each pair at Step 721involves applying a difference function of the representation of thebehavioral snapshot for two individuals. For example, in an embodimentusing vectors to store behaviors, each vector may be thought of asrepresenting a point in Euclidean space, and the distance between thepoints may be determined by the Euclidean distance. However, anydistance measure between two vectors can be used.

After associating each pair with the distance between individuals withinthe pair, in Step 722, the pair having the smallest distance isselected. This pair represents the two individuals having the mostsimilar behavior, and thus keeping both individuals contributes least todiversity of the candidate pool. The pair having the smallest distancecan be determined by sorting the pair list by distance. In Step 723 oneof the two individuals in the pair having the smallest distance isselected for discarding. There are several ways for determining which ofthe two individuals of the selected pair is the one to be selected fordiscarding. In one embodiment, one of the pair is selected at random. Inanother embodiment, the individual of the pair that has the closestneighbor outside the pair may be selected as the one to discard. Inother words, if a pair includes individuals I1 and I2, then a continuedsearch down the sorted list of pairs will find either another paircontaining I1 or another pair containing I2, first. Whichever is foundfirst is the individual which is to be selected for discarding.

In another embodiment, the fitness estimates of the individuals in thepair are compared and the individual having the lower fitness estimateis the one to be selected for discarding. However the individual isselected, the selected individual is removed from the candidate pool inStep 724.

In Step 725, the competition module determines whether any moreindividuals need to be discarded from the current experience layer ofthe candidate pool. Specifically, the remaining population of the layeris compared against the quota for the layer. If the quota is notexceeded, then no further elimination is required, and the process isdone. If the number of individuals in the layer still exceeds the quotafor the layer, then in step 726 all pairs that include the discardedindividual are removed from the list of pairs, and the process repeatswith the remaining pairs at Step 722.

Note that the above novelty competition is performed on all theindividuals in a layer of the elitist pool, which is a subset of all theindividuals in the overall candidate pool. In general, the noveltycompetition can be performed among all the individuals in at least anon-null subset of the individuals in the candidate pool. As usedherein, the term “subset” includes both proper and improper subsets, aswell as the null subset.

It can be seen that in competition module 514 in an evolutionary engine,the main criteria for competition is diversity of the behavior ofindividuals. Individuals with low fitness estimates often can be chosenover individuals with high fitness estimates, merely because the oneswith low fitness estimates are more novel than the ones with highfitness estimates. Fitness is used in the competition only in decidingwhich of two individuals with very similar behavior is to be selectedfor discarding—essentially in a tie breaker role. Thus individuals thatsurvive into the top layer in the evolutionary engine are notnecessarily the individuals having the greatest fitness estimates; theyare individuals having the greatest diversity of behavior.

EE—Harvesting

Returning to FIG. 6, in step 650, sometime after the top layer of thelocal candidate pool becomes full, the evolutionary engine harvestsindividuals from its top layer for forwarding to an evolutionarycoordinator. Candidate harvesting module 518 retrieves individuals forthat purpose. In one embodiment, candidate harvesting module 518retrieves individuals periodically, whereas in another embodiment itretrieves individuals only in response to user input. Preferably thecandidate harvesting module 518 maintains a list of individuals readyfor harvesting. It awakens periodically, and forwards all suchindividuals to the evolutionary coordinator. Preferably, candidateharvesting module 518 selects only from the layer or layers in the localcandidate pool 1132 whose minimum experience levels are at least as highas the minimum experience level of the lowest level (L₁) maintained bythe coordinator. Candidate harvesting module 518 also can apply furtherselection criteria as well in order to choose desirable individuals. Forexample, fitness might be used as a criterion at this stage of theprocess.

EE—Procreation

In Step 660, some individuals from the local candidate pool are used toprocreate new individuals. Only individuals in the local elitist pool(i.e. above layer L₀) are chosen for procreation. In an embodiment inwhich the elitist pool is populated based on relative novelty, a subsetof individuals selected as parents from that pool will be a diverse setof parents from which to procreate. In an embodiment, Procreation Module516 then selects the parent individuals randomly from the elitist pool.The parents may be selected from the most experienced layer of theelitist pool. Any conventional or future-developed technique can be usedfor procreation. In an embodiment, conditions, outputs, or rules fromparent individuals are combined in various ways to form childindividuals, and then, occasionally, they are mutated. The combinationprocess for example may include crossover—i.e., exchanging conditions,outputs, or entire rules between parent individuals to form childindividuals. New individuals created through procreation begin with anexperience level of zero, a fitness estimate that is undefined, and withno behavioral history. These individuals are placed in L₀ of the localcandidate pool 1132 in Step 610. In an embodiment, Procreation Module516 adds the new individuals into layer 0 of the candidate pooldirectly. In another embodiment, Procreation Module 516 provides the newindividuals to the Candidate Insertion Module 522.

Preferably, after new individuals are created by combination and/ormutation, the parent individuals are retained. In this case the parentindividuals also retain their experience level, their behavioralsnapshots, and their fitness estimates, and remain in their then-currentlocal elitist pool layers. In another embodiment, the parent individualsare discarded. After procreation, candidate testing module 512 operatesagain on the updated candidate pool. The process continues repeatedly.

Evolutionary Coordinators (EC)

As mentioned, the term “evolutionary coordinator” includes bothmid-chain evolutionary coordinators and the top-chain evolutionarycoordinator. An evolutionary coordinator coordinates the testing andcompetition of individuals generated by one or more evolutionaryengines. FIG. 8 is a block diagram showing structure of an evolutionarycoordinator, according to an embodiment of the invention, and FIG. 9 isa flow diagram illustrating the operations performed by an evolutionarycoordinator. The structure for a coordinator is similar to the structureof an evolutionary engine, though there are differences. An evolutionarycoordinator performs a subset of the operations described above for anevolutionary engine. An evolutionary coordinator does not create or testindividuals. A coordinator only compares individuals harvested from oneor more evolutionary engines to select the best performing individualsacross the evolutionary engines which it coordinates. The candidate poolfor a coordinator may not have a layer 0 because only individuals froman elitist pool in a down-chain unit are promoted to a coordinator. AnEC may use a different competition algorithm than that used in anevolutionary engine.

EC Receiving Individuals

In Step 910, new or returning individuals are received. For individualsnew to the coordinator's candidate pool 1122 that have been harvestedfrom a down-chain evolutionary unit, the aggregation module 816 recordsthe individual's fitness estimate. Returning individuals that hadpreviously been in the coordinator's candidate pool and were sent to adown-chain unit for more testing then sent back to the coordinator arereceived by the aggregation module 816, which updates the coordinator'sview of the fitness estimate based on the recent testing results.

EC—Competition

In Step 930 a competition module 814 of the evolutionary coordinatorassigns incoming individuals to a layer in the EC's local candidate poolbased on the individual's experience level. In an embodiment, anindividual returning to the coordinator after having been further testedmay not have had its experience level increased sufficiently for it toqualify for a higher experience layer, in which case the individual isnot yet considered for advancement within the local candidate pool. Onthe other hand, if the returned individual has had its experience levelincreased significantly, it may be considered for a layer that is twolayers or more above the individual's prior layer in the local candidatepool.

In an embodiment, Step 940 of deciding to discard an individual mayoccur in conjunction with assigning the individual to a layer. Thus,selecting individuals for discarding may be performed in a different waythan described above for the evolutionary engine. The fitness of theindividual may also be considered in discarding decisions. FIG. 10illustrates a flow diagram showing the operations performed fordiscarding excess individuals in an evolutionary coordinator. In step1010, the competition module stratifies individuals in the candidatepool in the EC into experience layers. In Step 1011 individuals havingless experience than the minimum experience required to enter the elitepool are assigned to Layer 0. In step 1012, the individuals in eachlayer are ranked based on their fitness estimate. In Step 1014,individuals that have achieved at least the minimum experience requiredto be in the top layer of the elitist pool, keep only the quota numberof the top ranked individuals. Individuals assigned to the top layer butdon't make the cut are discarded in Step 1020.

A minimum fitness for the elitist pool may be initialized to zero. Oncethe top layer has reached its quota of individuals, the minimum fitnessis assigned equal to the fitness of the least fit individual in the toplayer. This minimum fitness may change as the population of the toplayer changes. In Step 1016, individuals being promoted from Layer 0into the first layer of the elitist pool may only enter the elitist poolif their fitness estimate is at least as good as the minimum fitness.Individuals with enough experience to qualify for the elitist pool arenevertheless discarded if their fitness estimate doesn't reach therequired minimum. In an embodiment, same is true of individuals beingseen by the EC for the first time.

In Step 1018, for all other layers, (L₁ through L_(T−1)), keep withineach layer the quota number of top individuals ranked by fitnessestimate, and in Step 1020, discard the individuals in each layer thatdo not make the cut.

Thus in the embodiment of FIG. 10, in an evolutionary coordinator,individuals are discarded in dependence upon their fitness, and nottheir novelty.

EC—Sending Individuals for Further Testing

Returning to FIG. 9, In Step 950, individuals are selected forharvesting and individuals are selected for further testing. Acoordinator includes a candidate delegation module 812 which forwardsselected individuals from the coordinator's candidate pool a down-chainunit (which may be to an evolutionary engine) for further testing. Whenindividuals are sent to an evolutionary engine for further testing, theindividual also continues to compete based on its experience level andfitness within the evolutionary coordinator. If the individual isdiscarded by the coordinator during the time it is undergoing furthertesting, the updated experience and fitness information reported by thedown-chain unit if and when the individual is returned, are ignored.

EC—Harvesting

Sometime after the top layer of the local candidate pool 1122 is full,candidate harvesting module 818 selects individuals to be forwarded toan up-chain evolutionary coordinator or for use in production. In oneembodiment, individuals may be harvested from the entire elitist pool.In another embodiment, candidate harvesting module 818 selects only fromthe top layer L_(T) in the local candidate pool 1122, and can applyfurther selection criteria as well in order to choose desirableindividuals. For example, it can select only the fittest individualsfrom L_(T), and/or only those individuals that have shown lowvolatility. Other criteria will be apparent to the reader. Such criteriais usually specific to the application environment. The individuals alsomay undergo further validation as part of this further selectioncriteria, by testing on historical data not part of training data 114.The individuals selected by the candidate harvesting module 818 areremoved from the local candidate pool and forwarded to the up-chainevolutionary coordinator (if the EC is a mid-chain EC) or written to theproduction population database 122 for use by production system 112 aspreviously described (if the EC is the top-chain EC).

FIG. 12 illustrates various modules that can be used to implement thefunctionality of a mid-chain evolutionary coordinator of FIG. 11. Amid-chain coordinator has a similar structure to the coordinatorillustrated in FIG. 8. The candidate delegation module 1222 and thecandidate harvesting module 1218 may be the same as those described inFIG. 8. However, the candidate harvesting module 1218 for a mid-chaincoordinator does not place individuals into production, but insteadharvests individuals to send to another mid-chain coordinator or atop-chain coordinator.

As explained for a coordinator in FIG. 8, an aggregation module 1216records and/or updates an incoming individual's fitness estimate anddiscards individuals from its local candidate pool that were discardedby an EE after testing. If the incoming individual is new to thecoordinator and harvested from an engine down-chain, the incomingindividual may compete for acceptance into the EC 1120's local candidatepool 1122. The competition is performed by competition module 1214. Asfor the evolutionary engines 1130, the competition module 1214 alsoconsiders individuals from lower layers for promotion into higher layersin the local candidate pool 1122. Local candidate pool 1122 is updatedwith the revised contents.

Evolutionary coordinator 1120 also receives candidate individuals fromits up-chain evolutionary coordinator 1110 or 1120 for further testing.These individuals are received by a candidate insertion module 1222 inthe mid-chain EC 1120, but unlike when an evolutionary engine 1130receives a new candidate, no testing of the new candidate will beperformed on the coordinator itself, so the experience and fit estimateof the candidate is not subject to change. As a result, theseindividuals compete for entry into the EC's local candidate pool 1122based on their associated experience and fitness. Received individualsarrive in conjunction with both their fitness estimates and theirtesting experience levels, and compete for entry into the EC 1120'slocal candidate pool 1122 against only those individuals which occupythe same experience layer in the local candidate pool 1122. Thecandidate insertion module 1222 also takes a snapshot of the receivedindividuals for returning to the up-chain engine if and when it returnsthe individual after testing. As for the evolutionary engines 1130, thereceived candidates retain their experience level and fitness estimatesfrom above.

If one of the evolutionary engines 1120 or 1130 receives from itsup-chain EC 1110 or 1120, an individual for evaluation which it isalready in the process of evaluating, then the receiving evolutionaryengine it simply ignores the delegation. The receiving engine knows whatindividuals it is evaluating because it maintains a list of them, andwhere they came from, even if it has since further delegated evaluationto other engines down-chain. Though the receiving engine has been toldtwice to evaluate the individual, the up-chain requestor will not beconfused by receiving only one resulting report. The engine's reportinforms the up-chain requestor not only of the engine's testing results,but also the number of trials that the individual underwent under thecontrol of the engine, and this information is used in the mergingprocess performed by the requesting engine.

Sometime after the top layer of the local candidate pool 1122 is full,individuals can be harvested for forwarding to the EC's own up-chain EC.Candidate harvesting module 1218 retrieves individuals for that purpose.Preferably the candidate harvesting module 1218 maintains a list ofindividuals ready for reporting up. It awakens periodically, andforwards all the individuals on the list up-chain. As mentioned,candidate harvesting module 1218 preferably selects only from the layeror layers in the local candidate pool 1122 whose minimum experiencelevels are at least as high as the minimum experience level of thelowest layer (L₁) maintained by the immediately up-chain EC 1110 or 1120(or only from among those individuals with at least as high anexperience level). Candidate harvesting module 1218 also can applyfurther selection criteria as well in order to choose desirableindividuals. If the individuals had previously been received from theup-chain EC for testing, then candidate harvesting module 1218 alsoforwards the snapshot that it took of the individual upon receipt.

FIG. 13 illustrates various modules that can be used to implement thefunctionality of a top-chain evolutionary coordinator 1110. Thetop-chain EC's local candidate pool 1112 is also shown in the drawing,as is the production population database 122. Most of the modules shownin FIG. 13 can in some embodiments operate asynchronously from eachother.

As with the evolutionary engines 1130 and mid-chain evolutionarycoordinators 1120, the top-chain evolutionary coordinator 1110 alsoimplements a local layered candidate pool as described above withrespect to FIG. 2. Again the number of layers in the elitist pool, andthe minimum and maximum experience levels of such layers, need not bethe same as any or all of the down-chain engines. Preferably, though,they span a generally higher set of experience levels than all theimmediately down-chain engines. Like the mid-chain EC's 1120, top-chainEC 1110 does not maintain any individuals in L₀, though it does preventfurther testing of individuals in its top layer L_(T).

More specifically, the local candidate pool 1112 has multiple experiencelayers from its lowers layer L₁ to its highest layer L_(T). Typically L₁of the top-chain EC 1110 has a testing experience range who's minimumexperience level is higher than that of L₁ of each of the mid-chain EC's1120, though it could be equal in another embodiment. Individuals areharvested from only L_(T) of the top-chain EC 1110.

The modules in the top-chain evolutionary coordinator 1110 are similarto those in the mid-chain EC's 1120, except there is no candidateinsertion module for inserting any individuals received from anyup-chain neighbor. Instead, all individuals in the local candidate pool1112 were reported up from below.

Referring to FIG. 13, the candidate pool 1112 receives individuals fromthe EC's down-chain engines. Top-chain evolutionary coordinator 1110does not perform any of its own testing of candidate individuals, butinstead coordinates the testing performed by the down-chain engines.Thus top-chain EC 1110 includes a candidate delegation module 1312 whichselects individuals from its local candidate pool 1112 for furthertesting. The candidate delegation module 1312 selects the individualsusing any algorithm which tries to increase the experience level of allthe individuals in the local candidate pool 1112 other than those in thetop layer L_(T). The candidate delegation module 1312 does not need toactively load-balance its down-chain engines, since it only sendsindividuals down to a down-chain engine in response to a request fromthe down-chain engine for more individuals to test.

Candidates being reported up from below are received by an aggregationmodule 1316. Similarly as described above for the mid-chain engines1120, once the top-chain evolutionary coordinator 1110 sends a candidateto a down-chain engine, the down-chain engine is required to report itback, even if the candidate failed a competition below and wasdiscarded. Thus candidates received by aggregation module 1316 areeither individuals that failed below, in which case the top-chain EC1110 discards the individual from its own local candidate pool 1112, orindividuals that survived their tests below and are among the fittestindividuals that were in the down-chain engine's local candidate pool.Of the latter type, some may be returns of individuals that thetop-chain EC 1110 had previously sent down for further testing, andothers may have originated from a down-chain EE 1130. If an individualis a return of one that the top-chain EC 1110 had previously sent downfor further testing, then the aggregation module 1316 aggregates thecontribution that such further testing makes to the overall EC-centricfitness estimate before considering it for acceptance in to thetop-chain EC 1110's local candidate pool 1112. The aggregationmethodology described above for the mid-chain EC's 1120 can be used forthe top-chain EC 1110 as well.

If the returned individual is either a new individual that originatedbelow, or a returned individual that is proposed for acceptance into thetop-chain EC 1110's local candidate pool 1112, the individual isrequired to compete for its place in the EC 1110's local candidate pool1112. The competition is performed by competition module 1314. As forthe evolutionary engines 1130 and mid-chain evolutionary coordinators1120, the competition module 1314 considers individuals from lowerlayers for promotion into higher layers in the local candidate pool1112. An evolutionary coordinator 1120 may discard individuals that donot meet the minimum individual fitness of their target layer, anddiscard individuals that have been replaced in a layer by new entrantsinto that layer. Local candidate pool 1112 is updated with the revisedcontents.

Note that because the evolutionary engines 1130 are volunteercontributors to the system, they may go offline or lose communicationwith their up-chain engines at any time. This may also be true of somemid-chain EC's 1120 in some embodiments. Thus it is possible that someindividuals that an EC 1110 or 1120 sent down-chain for further testingwill never be returned to the sending EC. In this case the prior copy ofthe individual, retained by the EC, remains in place in its localcandidate pool unless and until it is displaced through competition inthe EC. Still further, note that an individual retained in an EC afterit has also been sent to a down-chain engine for further testing, maybecome displaced and discarded from the EC through competition in theEC. In this case, if the same individual is returned by the down-chainengine, the EC simply ignores it.

Example Sequence

Given the above principles, the following is an example sequence ofsteps that might occur in the arrangement of FIG. 11 as individuals arecreated, tested, subjected to competition, evolved, and eventuallyharvested. Many steps are omitted as the system operates on numerousindividuals and numerous evolutionary not mentioned herein. Many stepsare omitted also in between the steps set forth, for purposes ofclarity. In addition, for purposes of clarity several of theevolutionary engines in FIG. 11 are referred to by shorthandabbreviations EC1, EC2, EC4, EE2, EE3, EE4, EE5, EE6 and TEC, all asindicated in FIG. 11.

EE2 creates candidates, including Individual #1, writes to localcandidate pool

EE2 tests the candidates in local candidate pool, including discardingsome through local competition, procreating to make new candidates, andcreating new candidates randomly. The local competition in EE2specifically favors novelty of individuals, not fitness, though fitnessmay be used in the so-called tie-breaker role mentioned above.

Individual #1 reaches top layer in local candidate pool

EE2 transmits candidates from top layer, including Individual #1 andEE2's view of Individual #1's fitness level, to mid-chain EC1

EC1 accepts Individual #1 after competition against other candidates inEC1's local candidate pool. EC1's view of Individual #1's fitness levelis now equal to EE2's view of Individual #1's fitness level. EC1 writesIndividual #1 into L1 of local candidate pool with EC1's view ofIndividual #1's fitness level. Competition in EC1 favors fitness ofindividuals.

EC1 receives request from EE2 for candidates to test.

EC1 transmits candidates, including Individual #1, to EE2 for furthertesting.

EE2 inserts Individual #1 into EE2's local candidate pool.

EE2 tests the candidates in its local candidate pool, includingIndividual #1, including discarding some through local competition(favoring novelty), procreating to make new candidates, and creating newcandidates randomly. Individual #1 survives the competition.

Before receiving back Individual #1 from EE2, EC1 receives request fromEE3 for candidates to test.

EC1 transmits candidates, again including Individual #1, to EE3 forfurther testing.

EE3 inserts Individual #1 into EE3's local candidate pool.

EE3 tests the candidates in its local candidate pool, includingIndividual #1, including discarding some through local competition(favoring novelty), procreating to make new candidates, and creating newcandidates randomly. Individual #1 survives.

Individual #1 reaches top layer in EE2's local candidate pool

EE2 transmits candidates from top layer, including Individual #1, to EC1with its own view of Individual #1's updated fitness level.

EC1 accepts Individual #1 after competition (favoring fitness) againstother candidates in EC1's local candidate pool. Writes Individual #1into experience-appropriate layer of local candidate pool. Merges EE2'sview of Individual #1's fitness level with EC1's view and writes updatedview of Individual #1's fitness level into EC1's local candidate pool.

Individual #1 reaches top layer in EE3's local candidate pool

EE3 transmits candidates from top layer, including Individual #1, to EC1with its own view of Individual #1's updated fitness level.

EC1 accepts Individual #1 after competition (favoring fitness) againstother candidates in EC1's local candidate pool. Writes Individual #1into experience-appropriate layer of local candidate pool. Merges EE3'sview of Individual #1's fitness level with EC1's view and writes updatedview of Individual #1's fitness level into EC1's local candidate pool.

EC1 sends request to top-chain TEC for candidates to test.

TEC transmits candidates, including Individual #2, to EC1 for furthertesting.

EC1 accepts Individual #2 after competition (favoring fitness) againstother candidates in EC1's local candidate pool.

EC1 continues to coordinate further testing of the candidates in itslocal candidate pool, including Individual #1 and Individual #2,including delegating testing of Individual #1 and/or Individual #2 toEE2 and/or EE3, receiving them back after testing with new fitnessestimates as viewed by EE2 and/or EE3, and discarding some through localcompetition (favoring fitness) with other candidates in EC1's localcandidate pool.

Individual #1 and Individual #2 reach top layer in EC1's local candidatepool.

EC1 transmits candidates from top layer, including Individual #1 andIndividual #2, to TEC with EC1's view of Individual #1's and Individual#2's updated fitness levels.

TEC accepts Individual #1 and Individual #2 after competition (favoringfitness) against other candidates in TEC local candidate pool. WritesIndividual #1 and Individual #2 into L1 of local candidate pool. MergesEC1's view of Individual #2's fitness level with TEC's view and writesupdated view of Individual #2's fitness level into TEC's local candidatepool. Since Individual #1 is new to TEC, TEC's view of Individual #1'sfitness level is now equal to EC1's view of Individual #1's fitnesslevel.

Mid-chain EC2 sends request to top-chain TEC for candidates to test.

TEC transmits candidates, including Individual #1, to EC2 for furthertesting.

EC2 accepts Individual #1 after competition (favoring fitness) againstother candidates in EC2's local candidate pool.

Mid-chain EC4 sends request to EC2 for candidates to test.

EC2 transmits candidates, including Individual #1, to EC4 for furthertesting.

EC4 accepts Individual #1 after competition (favoring fitness) againstother candidates in EC4's local candidate pool.

EE5 sends request to EC4 for candidates to test.

EE5 transmits candidates, including Individual #1, to EE5 for furthertesting.

EE5 inserts Individual #1 into EE5's local candidate pool.

EE5 tests the candidates in its local candidate pool, includingIndividual #1, including discarding some through local competition(favoring novelty), procreating to make new candidates, and creating newcandidates randomly

Individual #1 reaches top layer in EE5's local candidate pool

EE5 transmits candidates from top layer, including Individual #1, to EC4with its own view of Individual #1's updated fitness level.

EC4 accepts Individual #1 after competition (favoring fitness) againstother candidates in EC4's local candidate pool. Merges EE5's view ofIndividual #1's fitness level with EC4's view and writes updated view ofIndividual #1's fitness level into EC4's local candidate pool.

EC4 continues to coordinate further testing of the candidates in itslocal candidate pool, including Individual #1, including delegatingtesting of Individual #1 to EE5 and/or EE6, receiving them back aftertesting with new fitness estimates as viewed by EE5 and/or EE6, anddiscarding some through local competition (favoring fitness) with othercandidates in EC4's local candidate pool.

Individual #1 reaches top layer in EC4's local candidate pool.

EC4 transmits candidates from top layer, including Individual #1, to EC2with EC4's view of Individual #1's updated fitness levels.

EC2 accepts Individual #1 after competition (favoring fitness) againstother candidates in EC2's local candidate pool. Writes Individual #1into appropriate layer of local candidate pool. Merges EC4's view ofIndividual #1's fitness level with EC2's view and writes updated view ofIndividual #1's fitness level into EC2's local candidate pool.

EC2 continues to coordinate further testing of the candidates in itslocal candidate pool, including Individual #1, including delegatingtesting of Individual #1 to EE4 and/or EC4, receiving them back aftertesting with new fitness estimates as viewed by EE4 and/or EC4, anddiscarding some through local competition (favoring fitness) with othercandidates in EC2's local candidate pool.

EC2 transmits candidates from top layer, including Individual #1, to TECwith EC2's view of Individual #1's updated fitness levels.

TEC accepts Individual #1 after competition (favoring fitness) againstother candidates in TEC's local candidate pool. Writes Individual #1into experience-appropriate layer of local candidate pool. Merges EC2'sview of Individual #1's fitness level with TEC's view and writes updatedview of Individual #1's fitness level into TEC's local candidate pool.

Individual #1 reaches top layer in TEC's local candidate pool.

Individual #1 is harvested for production population.

It will be appreciated that whereas in embodiments described herein onlythe evolutionary engines cause their individuals to compete on noveltywhereas all of the evolutionary coordinators cause their individuals tocompete on fitness, in another embodiment some of the evolutionarycoordinators can cause their individuals to compete at least in partbased on novelty as well.

Computer Hardware

FIG. 14 is a simplified block diagram of a computer system 1410 that canbe used to implement any or all of the evolutionary engines 1110, 1120,and 1130, the production system 112, and the data feed server 1140.While FIGS. 5, 8, 12, and 13 indicate individual components for carryingout specified operations, it will be appreciated that each componentactually causes a computer system such as 1410 to operate in thespecified manner.

Computer system 1410 typically includes a processor subsystem 1414 whichcommunicates with a number of peripheral devices via bus subsystem 1412.These peripheral devices may include a storage subsystem 1424,comprising a memory subsystem 1426 and a file storage subsystem 1428,user interface input devices 1422, user interface output devices 1420,and a network interface subsystem 1416. The input and output devicesallow user interaction with computer system 1410. Network interfacesubsystem 1416 provides an interface to outside networks, including aninterface to communication network 1418, and is coupled viacommunication network 1418 to corresponding interface devices in othercomputer systems. For the evolutionary engines 1110, 1120, and 1130,communication with the engine's up-chain and down-chain engines occursvia communication network 1418. Communication network 1418 may comprisemany interconnected computer systems and communication links. Thesecommunication links may be wireline links, optical links, wirelesslinks, or any other mechanisms for communication of information. Whilein one embodiment, communication network 1418 is the Internet, in otherembodiments, communication network 1418 may be any suitable computernetwork or combination of computer networks.

The physical hardware component of network interfaces are sometimesreferred to as network interface cards (NICs), although they need not bein the form of cards: for instance they could be in the form ofintegrated circuits (ICs) and connectors fitted directly onto amotherboard, or in the form of macrocells fabricated on a singleintegrated circuit chip with other components of the computer system.

User interface input devices 1422 may include a keyboard, pointingdevices such as a mouse, trackball, touchpad, or graphics tablet, ascanner, a touch screen incorporated into the display, audio inputdevices such as voice recognition systems, microphones, and other typesof input devices. In general, use of the term “input device” is intendedto include all possible types of devices and ways to input informationinto computer system 1410 or onto computer network 1418.

User interface output devices 1420 may include a display subsystem, aprinter, a fax machine, or non-visual displays such as audio outputdevices. The display subsystem may include a cathode ray tube (CRT), aflat-panel device such as a liquid crystal display (LCD), a projectiondevice, or some other mechanism for creating a visible image. Thedisplay subsystem may also provide non-visual display such as via audiooutput devices. In general, use of the term “output device” is intendedto include all possible types of devices and ways to output informationfrom computer system 1410 to the user or to another machine or computersystem. In particular, an output device of the computer system 1410 onwhich production system 112 is implemented, may include a visual outputinforming a user of action recommendations made by the system, or mayinclude a communication device for communicating action signals directlyto the controlled system 128. Additionally or alternatively, thecommunication network 1418 may communicate action signals to thecontrolled system 128. In the financial asset trading environment, forexample, the communication network 1418 transmits trading signals to acomputer system in a brokerage house which attempts to execute theindicated trades.

Storage subsystem 1424 stores the basic programming and data constructsthat provide the functionality of certain embodiments of the presentinvention. For example, the various modules implementing thefunctionality of certain embodiments of the invention may be stored instorage subsystem 1424. These software modules are generally executed byprocessor subsystem 1414. Storage subsystem 1424 also stores thecandidate pools 1112, 1122, or 1132, as the case may be, for arespective evolutionary engine. For the data feed 1140 storage subsystem1424 may store the training database 114. For the top-chain EC 1110and/or for production system 112, storage subsystem 1424 may store theproduction population 122. Alternatively, one or more of such databasescan be physically located elsewhere, and made accessible to the computersystem 1410 via the communication network 1418.

Memory subsystem 1426 typically includes a number of memories includinga main random access memory (RAM) 1430 for storage of instructions anddata during program execution and a read only memory (ROM) 1432 in whichfixed instructions are stored. File storage subsystem 1428 providespersistent storage for program and data files, and may include a harddisk drive, a floppy disk drive along with associated removable media, aCD ROM drive, an optical drive, or removable media cartridges. Thedatabases and modules implementing the functionality of certainembodiments of the invention may have been provided on a computerreadable medium such as one or more CD-ROMs, and may be stored by filestorage subsystem 1428. The host memory 1426 contains, among otherthings, computer instructions which, when executed by the processorsubsystem 1414, cause the computer system to operate or performfunctions as described herein. As used herein, processes and softwarethat are said to run in or on “the host” or “the computer”, execute onthe processor subsystem 1414 in response to computer instructions anddata in the host memory subsystem 1426 including any other local orremote storage for such instructions and data.

As used herein, a computer readable medium is one on which informationcan be stored and read by a computer system. Examples include a floppydisk, a hard disk drive, a RAM, a CD, a DVD, flash memory, a USB drive,and so on. The computer readable medium may store information in codedformats that are decoded for actual use in a particular data processingsystem. A single computer readable medium, as the term is used herein,may also include more than one physical item, such as a plurality of CDROMs or a plurality of segments of RAM, or a combination of severaldifferent kinds of media. As used herein, the term does not include meretime varying signals in which the information is encoded in the way thesignal varies over time.

Bus subsystem 1412 provides a mechanism for letting the variouscomponents and subsystems of computer system 1410 communicate with eachother as intended. Although bus subsystem 1412 is shown schematically asa single bus, alternative embodiments of the bus subsystem may usemultiple busses.

Computer system 1410 itself can be of varying types including a personalcomputer, a portable computer, a workstation, a computer terminal, anetwork computer, a television, a mainframe, a server farm, awidely-distributed set of loosely networked computers, or any other dataprocessing system or user device. Due to the ever-changing nature ofcomputers and networks, the description of computer system 1410 depictedin FIG. 14 is intended only as a specific example for purposes ofillustrating the preferred embodiments of the present invention. Manyother configurations of computer system 1410 are possible having more orless components than the computer system depicted in FIG. 14.

As used herein, a given signal, event or value is “responsive” to apredecessor signal, event or value if the predecessor signal, event orvalue influenced the given signal, event or value. If there is anintervening processing element, step or time period, the given signal,event or value can still be “responsive” to the predecessor signal,event or value. If the intervening processing element or step combinesmore than one signal, event or value, the signal output of theprocessing element or step is considered “responsive” to each of thesignal, event or value inputs. If the given signal, event or value isthe same as the predecessor signal, event or value, this is merely adegenerate case in which the given signal, event or value is stillconsidered to be “responsive” to the predecessor signal, event or value.“Dependency” of a given signal, event or value upon another signal,event or value is defined similarly.

Applicants hereby disclose in isolation each individual featuredescribed herein and each combination of two or more such features, tothe extent that such features or combinations are capable of beingcarried out based on the present specification as a whole in light ofthe common general knowledge of a person skilled in the art,irrespective of whether such features or combinations of features solveany problems disclosed herein, and without limitation to the scope ofthe claims. Applicants indicate that aspects of the present inventionmay consist of any such feature or combination of features. In view ofthe foregoing description it will be evident to a person skilled in theart that various modifications may be made within the scope of theinvention.

The foregoing description of preferred embodiments of the presentinvention has been provided for the purposes of illustration anddescription. It is not intended to be exhaustive or to limit theinvention to the precise forms disclosed. Obviously, many modificationsand variations will be apparent to practitioners skilled in this art. Inparticular, and without limitation, any and all variations described,suggested or incorporated by reference in the Background section or theCross References section of this patent application are specificallyincorporated by reference into the description herein of embodiments ofthe invention. For example, though embodiments described herein useexperience-layered candidate pools, candidate pools used in otherembodiments need not be experience-layered. In addition, any and allvariations described, suggested or incorporated by reference herein withrespect to any one embodiment are also to be considered taught withrespect to all other embodiments. The embodiments described herein werechosen and described in order to best explain the principles of theinvention and its practical application, thereby enabling others skilledin the art to understand the invention for various embodiments and withvarious modifications as are suited to the particular use contemplated.It is intended that the scope of the invention be defined by thefollowing claims and their equivalents.

We claim:
 1. An arrangement of processing units, for use with a datamining training database containing training data, comprising: a firstprocessing unit having a first processor and a first memory, the firstmemory storing a first local pool of candidate individuals, each of thecandidate individuals in the first local pool identifying at least oneclassification rule and further identifying a respective fitnessestimate centric to the first processing unit; and a second processingunit disposed down-chain of the first processing unit, the secondprocessing unit having a second processor and a second memory, thesecond memory storing a second local pool of candidate individuals, eachof the candidate individuals in the second local pool identifying atleast one classification rule and further identifying a respectivefitness estimate centric to the second processing unit, wherein thefirst processor is configured to: store in the first memory individualsfor evaluation against portions of the training data, delegate, to thesecond processing unit, evaluation of individuals from the first localpool of candidate individuals, update the fitness estimates, centric tothe first processing unit, for selected ones of individuals receivedback from the second processing unit after testing, in dependence uponresults of such testing, and select individuals for discarding from thefirst local pool, in dependence upon their fitness estimates relative tothe fitness estimates of the other individuals within the first localpool, and wherein the second processor is configured to: store in thesecond local pool of candidate individuals, individuals received by thesecond processing unit from the first processing unit for evaluation,test individuals from the second local pool of candidate individualsagainst a portion of the training data, wherein testing individuals fromthe second local pool of candidate individuals against a portion of thetraining data, includes recording, for each individual tested, anidentification of the behavior of the individual when tested against theportion of the training data, and wherein selecting individuals fordiscarding from the second local pool in dependence upon their noveltyrelative to other individuals in the second local pool comprisesiteratively: identifying, among the individuals in at least a non-nullsubset of the individuals in the second local pool, the pair whoserecorded behavior is most similar; and selecting one individual of theidentified pair for discarding, update the fitness estimates, centric tothe second processing unit, for selected ones of individuals tested, independence upon the results of such testing, select individuals fordiscarding from the second local pool, in dependence upon their noveltyrelative to other individuals in the second local pool, and report tothe first processing unit, individuals from the second local pool andnot selected for discarding, in conjunction with the individuals'fitness estimates, centric to the second processing unit.
 2. Thearrangement of claim 1, wherein selecting individuals for discardingfrom the second local pool in dependence upon their novelty relative toother individuals in the second local pool, is performed further independence upon their fitness estimates relative to other individuals inthe second local pool.
 3. The arrangement of claim 1, wherein selectingone individual of the identified pair for discarding comprises selectingfor discarding the individual of the identified pair having a lowerfitness estimate, centric to the second processing unit.
 4. Thearrangement of claim 1, wherein the second processor is furtherconfigured to add into the second local pool individuals developed byreproduction in dependence upon parent individuals remaining in thesecond local pool after the selecting individuals for discarding fromthe second local pool.
 5. A computer system which transmits toward asecond processing unit, software which, when executed by a secondprocessor performs: storing in a second local pool of candidateindividuals, individuals received by the second processing unit from afirst processing unit for evaluation, testing individuals from thesecond local pool of candidate individuals against a portion of trainingdata, updating fitness estimates, centric to the second processing unit,for selected ones of individuals tested, in dependence upon results ofsuch testing, selecting individuals for discarding from the second localpool, in dependence upon their novelty relative to other individuals inthe second local pool, wherein selecting individuals for discarding fromthe second local pool in dependence upon their novelty relative to otherindividuals in the second local pool, is performed further in dependenceupon their fitness estimates relative to other individuals in the secondlocal pool, and reporting toward the first processing unit, individualsfrom the second local pool and not selected for discarding, inconjunction with the individuals' fitness estimates, centric to thesecond processing unit.
 6. The computer system of claim 5, which furthertransmits toward the first processing unit having a first processor,software which, when executed by the first processor performs: storingin a first memory individuals for evaluation against portions of thetraining data, delegating, to the second processing unit, evaluation ofindividuals from a first local pool of candidate individuals, updatingfitness estimates, centric to the first processing unit, for selectedones of individuals received back from the second processing unit aftertesting, in dependence upon the results of such testing, and selectingindividuals for discarding from the first local pool, in dependence upontheir fitness estimates relative to the fitness estimates of the otherindividuals within the first local pool.
 7. The computer system of claim5, wherein testing individuals from the second local pool of candidateindividuals against a portion of the training data, includes recording,for each individual tested, an identification of the behavior of theindividual when tested against the portion of the training data, andwherein selecting individuals for discarding from the second local poolin dependence upon their novelty relative to other individuals in thesecond local pool comprises iteratively: identifying, among theindividuals in at least a non-null subset of the individuals in thesecond local pool, the pair whose recorded behavior is most similar; andselecting one individual of the identified pair for discarding.
 8. Thecomputer system of claim 7, wherein selecting one individual of theidentified pair for discarding comprises selecting for discarding theindividual of the identified pair having a lower fitness estimate,centric to the second processing unit.
 9. The computer system of claim5, wherein the second processor is further configured to add into thesecond local pool individuals developed by reproduction in dependenceupon parent individuals remaining in the second local pool after theselecting individuals for discarding from the second local pool.
 10. Anevolutionary engine for use with a data mining training databasecontaining training data, and for use further with an evolutionarycoordinator having a pool of candidate individuals stored accessiblythereto, the evolutionary engine comprising: a processor and a memory,the memory storing accessibly to the processor a local pool of candidateindividuals, each of the candidate individuals in the local poolidentifying at least one classification rule and further identifying arespective fitness estimate centric to the evolutionary engine, whereinthe processor is configured to: store in the local pool, individualsreceived by the evolutionary engine from the evolutionary coordinatorfor evaluation, test individuals from the local pool against a portionof the training data, wherein in testing individuals from the local poolof candidate individuals against a portion of the training data, theprocessor is configured to record, for each individual tested, anidentification of the behavior of the individual when tested against theportion of the training data, and wherein in selecting individuals fordiscarding from the local pool in dependence upon their novelty relativeto other individuals in the local pool the processor is configured toiteratively: identify, among the individuals in at least a non-nullsubset of the individuals in the local pool, the pair whose recordedbehavior is most similar; and select one individual of the identifiedpair for discarding, update the fitness estimates, centric to theevolutionary engine, for selected ones of individuals tested, independence upon the results of such testing, select individuals fordiscarding from the local pool, in dependence upon their noveltyrelative to other individuals in the local pool, and report toward theevolutionary coordinator individuals from the local pool and notselected for discarding, in conjunction with the individuals' fitnessestimates centric to the evolutionary coordinator.
 11. The evolutionaryengine of claim 10, wherein in selecting individuals for discarding fromthe local pool in dependence upon their novelty relative to otherindividuals in the local pool, the processor is configured to select theindividuals for discarding further in dependence upon their fitnessestimates relative to other individuals in the local pool.
 12. Theevolutionary engine of claim 10, wherein in selecting one individual ofthe identified pair for discarding the processor is configured to selectfor discarding the individual of the identified pair having a lowerfitness estimate, centric to the evolutionary engine.
 13. Theevolutionary engine of claim 10, wherein the processor is furtherconfigured to add into the local pool individuals developed byreproduction in dependence upon parent individuals remaining in thelocal pool after the selecting individuals for discarding from the localpool.
 14. The evolutionary engine of claim 10, wherein in selectingindividuals for reporting, the processor is configured to select theindividuals for reporting in dependence upon the fitness estimates ofthe individuals centric to the evolutionary engine, from among theindividuals in the local pool not yet selected for discarding.